An obstacle to artificial general intelligence is set by the continual learning of multiple tasks of different nature. Recently, various heuristic tricks, both from machine learning and from neuroscience angles, were proposed, but they lack a unified theory ground. Here, we focus on the continual learning in single-layered and multi-layered neural networks of binary weights. A variational Bayesian learning setting is thus proposed, where the neural network is trained in a field-space, rather than the gradient-ill-defined discrete-weight space, and furthermore, the weight uncertainty is naturally incorporated, and modulates the synaptic resources among tasks. From a physics perspective, we translate the variational continual learning into the Franz-Parisi thermodynamic potential framework, where the previous task knowledge acts as a prior and a reference as well. Therefore, the learning performance can be analytically studied with mean-field order parameters, whose predictions coincide with the numerical experiments using stochastic gradient descent methods. Our proposed principled frameworks also connect to elastic weight consolidation, and neuroscience inspired metaplasticity, providing a theory-grounded method for the real-world multi-task learning with deep networks.

骨架序列是紧凑而轻巧的。已经提出了许多基于骨架的动作识别者来对人类行为进行分类。在这项工作中，我们旨在结合与现有模型兼容的组件，并进一步提高其准确性。为此，我们设计了两个时间配件：离散余弦编码（DCE）和按时间顺序损失（CRL）。DCE促进模型以分析频域的运动模式，同时减轻信号噪声的影响。CRL指导网络明确捕获序列的时间顺序。这两个组件一致地赋予许多最近提供的动作识别器具有准确性的提升，从而在两个大数据集上实现了新的最先进（SOTA）精度。

基于骨架的动作识别由于数据集的轻质，紧凑的性质，吸引了从业者和研究人员。与基于RGB视频的动作识别相比，基于骨架的动作识别是一种更安全的方法来保护受试者的隐私，同时具有竞争性识别性能。但是，由于骨架估计算法以及运动和深度传感器的改进，可以在骨架数据集中保留运动特性的更多细节，从而导致数据集的潜在隐私泄漏。要调查骨架数据集的潜在隐私泄漏，我们首先将分类器从关节的轨迹中分类敏感私人信息。实验表明，培训的模型对性别进行分类，可以预测88％的准确性，并重新识别具有82％的准确性的人。我们提出了两个匿名化算法的变体来保护骨架数据集的潜在隐私泄漏。实验结果表明，匿名数据集可以降低隐私泄漏的风险，同时对动作识别性能产生边际影响。

离散基因监管网络（GRNS）在鲁棒性和模块化的研究中起着至关重要的作用。评估GRNS稳健性的常见方法是测量它们调节一组扰动基因激活图案回到其未受干扰的形式的能力。通常，通过收集通过基因激活模式的预定分布产生的随机样品来获得扰动。这种采样方法引入了随机性，否定动态。这种动态施加在已经复杂的健身景观之上。因此，在使用采样的情况下，重要的是要理解哪种效果来自健身景观的结构，并且从施加的动力学产生。健身功能的随机性也会导致重现性和实验后分析中的困难。通过考虑基因活性模式的完全分布，我们制定确定性分布适应性评估，以避免适应性评估中的随机性。这种健身评估有助于重复性。其确定性允许我们在健身上确定理论界，从而确定算法是否达到了全局最优。它使我们能够将问题域与嘈杂的健身评估的影响区分开来，从而解决〜\ CiteT {espinosa2010Specialization}问题领域的行为中的两个剩余异常。我们还揭示了解决方案GRNS的一些属性，使它们具有稳健和模块化，导致对问题域的性质更深入了解。我们通过讨论潜在的方向来模拟和理解较大，更复杂的域中的模块化的出现，这是产生更有用的模块化解决方案的关键，并理解生物系统中的模块化的难以。

骨架序列轻巧且紧凑，因此是在边缘设备上进行动作识别的理想候选者。最新的基于骨架的动作识别方法从3D关节坐标作为时空提示提取特征，在图神经网络中使用这些表示形式来提高识别性能。一阶和二阶特征（即关节和骨骼表示）的使用导致了很高的精度。但是，许多模型仍然被具有相似运动轨迹的动作所困惑。为了解决这些问题，我们建议以角度编码为现代体系结构的形式融合高阶特征，以稳健地捕获关节和身体部位之间的关系。这种与流行的时空图神经网络的简单融合可在包括NTU60和NTU120在内的两个大型基准中实现新的最新精度，同时使用较少的参数和减少的运行时间。我们的源代码可公开可用：https：//github.com/zhenyueqin/angular-skeleton-soding。

This paper focuses on designing efficient models with low parameters and FLOPs for dense predictions. Even though CNN-based lightweight methods have achieved stunning results after years of research, trading-off model accuracy and constrained resources still need further improvements. This work rethinks the essential unity of efficient Inverted Residual Block in MobileNetv2 and effective Transformer in ViT, inductively abstracting a general concept of Meta-Mobile Block, and we argue that the specific instantiation is very important to model performance though sharing the same framework. Motivated by this phenomenon, we deduce a simple yet efficient modern \textbf{I}nverted \textbf{R}esidual \textbf{M}obile \textbf{B}lock (iRMB) for mobile applications, which absorbs CNN-like efficiency to model short-distance dependency and Transformer-like dynamic modeling capability to learn long-distance interactions. Furthermore, we design a ResNet-like 4-phase \textbf{E}fficient \textbf{MO}del (EMO) based only on a series of iRMBs for dense applications. Massive experiments on ImageNet-1K, COCO2017, and ADE20K benchmarks demonstrate the superiority of our EMO over state-of-the-art methods, \eg, our EMO-1M/2M/5M achieve 71.5, 75.1, and 78.4 Top-1 that surpass \textbf{SoTA} CNN-/Transformer-based models, while trading-off the model accuracy and efficiency well.

Supervised Question Answering systems (QA systems) rely on domain-specific human-labeled data for training. Unsupervised QA systems generate their own question-answer training pairs, typically using secondary knowledge sources to achieve this outcome. Our approach (called PIE-QG) uses Open Information Extraction (OpenIE) to generate synthetic training questions from paraphrased passages and uses the question-answer pairs as training data for a language model for a state-of-the-art QA system based on BERT. Triples in the form of <subject, predicate, object> are extracted from each passage, and questions are formed with subjects (or objects) and predicates while objects (or subjects) are considered as answers. Experimenting on five extractive QA datasets demonstrates that our technique achieves on-par performance with existing state-of-the-art QA systems with the benefit of being trained on an order of magnitude fewer documents and without any recourse to external reference data sources.

Transformer has achieved impressive successes for various computer vision tasks. However, most of existing studies require to pretrain the Transformer backbone on a large-scale labeled dataset (e.g., ImageNet) for achieving satisfactory performance, which is usually unavailable for medical images. Additionally, due to the gap between medical and natural images, the improvement generated by the ImageNet pretrained weights significantly degrades while transferring the weights to medical image processing tasks. In this paper, we propose Bootstrap Own Latent of Transformer (BOLT), a self-supervised learning approach specifically for medical image classification with the Transformer backbone. Our BOLT consists of two networks, namely online and target branches, for self-supervised representation learning. Concretely, the online network is trained to predict the target network representation of the same patch embedding tokens with a different perturbation. To maximally excavate the impact of Transformer from limited medical data, we propose an auxiliary difficulty ranking task. The Transformer is enforced to identify which branch (i.e., online/target) is processing the more difficult perturbed tokens. Overall, the Transformer endeavours itself to distill the transformation-invariant features from the perturbed tokens to simultaneously achieve difficulty measurement and maintain the consistency of self-supervised representations. The proposed BOLT is evaluated on three medical image processing tasks, i.e., skin lesion classification, knee fatigue fracture grading and diabetic retinopathy grading. The experimental results validate the superiority of our BOLT for medical image classification, compared to ImageNet pretrained weights and state-of-the-art self-supervised learning approaches.

Knowledge graph embedding (KGE), which maps entities and relations in a knowledge graph into continuous vector spaces, has achieved great success in predicting missing links in knowledge graphs. However, knowledge graphs often contain incomplete triples that are difficult to inductively infer by KGEs. To address this challenge, we resort to analogical inference and propose a novel and general self-supervised framework AnKGE to enhance KGE models with analogical inference capability. We propose an analogical object retriever that retrieves appropriate analogical objects from entity-level, relation-level, and triple-level. And in AnKGE, we train an analogy function for each level of analogical inference with the original element embedding from a well-trained KGE model as input, which outputs the analogical object embedding. In order to combine inductive inference capability from the original KGE model and analogical inference capability enhanced by AnKGE, we interpolate the analogy score with the base model score and introduce the adaptive weights in the score function for prediction. Through extensive experiments on FB15k-237 and WN18RR datasets, we show that AnKGE achieves competitive results on link prediction task and well performs analogical inference.

Digital engineering transformation is a crucial process for the engineering paradigm shifts in the fourth industrial revolution (4IR), and artificial intelligence (AI) is a critical enabling technology in digital engineering transformation. This article discusses the following research questions: What are the fundamental changes in the 4IR? More specifically, what are the fundamental changes in engineering? What is digital engineering? What are the main uncertainties there? What is trustworthy AI? Why is it important today? What are emerging engineering paradigm shifts in the 4IR? What is the relationship between the data-intensive paradigm and digital engineering transformation? What should we do for digitalization? From investigating the pattern of industrial revolutions, this article argues that ubiquitous machine intelligence (uMI) is the defining power brought by the 4IR. Digitalization is a condition to leverage ubiquitous machine intelligence. Digital engineering transformation towards Industry 4.0 has three essential building blocks: digitalization of engineering, leveraging ubiquitous machine intelligence, and building digital trust and security. The engineering design community at large is facing an excellent opportunity to bring the new capabilities of ubiquitous machine intelligence and trustworthy AI principles, as well as digital trust, together in various engineering systems design to ensure the trustworthiness of systems in Industry 4.0.